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Abstract. - We study the response of an attractor neural network, in the ferromagnetic 
phase, to an external, time-dependent stimulus, which drives the system periodically toward 
two different attractors. We demonstrate a non-trivial dependence of the response of the system 
via a system-size resonance, by showing a signal amplification maximum at a certain finite size. 



Introduction. The counter-intuitive role of fluctuations as a source of order has at- 
tracted much attention for the last years [1,2]. Particularly interesting is the phenomenon of 
stochastic resonance where the response of a non-linear system to the action of a weak signal 
is enhanced, not hindered, by the addition of an optimal amount of noise [3]. Among several 
potential applications of stochastic resonance, there is evidence that it plays an important 
role in some cognitive processes, such as perception [4-6]. 

Another phenomenon that recently appeared in the literature and is closely related to 
stochastic resonance is system size resonance, or SSR from now on [7]. In SSR, the presence 
of noise in a system of finite size, close to a second-order phase transition, gives rise to the 
appearance of an optimal size for the system to adapt to an external field [7-9] . 

Inspired by the applications of stochastic resonance to cognitive processes, in this Letter 
we show that SSR can operate in a simple model of associative memory, namely, a Hopfield 
neural network [10], improving its ability to follow a time-dependent stimulus. We will focus 
on the simplest case of a Hopfield network storing just two patterns. 

Model. - The model is defined by the following Hamiltonian [10]: 
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where Sj = ±1, i — 1, N are A-binary neurons and C* = ±1, i =,..., N, /J, = 1,2 are the 
two binary patterns the system is trained with. We assume that the network has been trained 
following the Hebb rule [11]: 

Jij (2) 

The second term in Eq. Q represents a non constant but periodic stimulus of period 2T: 
n(t) = 1 if t € [0, T) and n(t) — 2 if i G [T, 2T), which biases the system towards pattern 1 
and 2, alternatively. The intensity of the stimulus is given by h. 

A relevant magnitude in our analysis is the Hamming distance between the patterns: 



1 N 

rf =2^E^-^i' (3) 



i.e., the fraction of sites in which both patterns are different. It varies between 0, when both 
patterns are equal at every site, and 1 when they are completely different. 

This system has been extensively studied in the literature, mainly in the thermodynamic 
limit N — > oo (see [12, 13] and the references therein). The role of finite size has only been 
studied in order to find finite-size corrections to infinite-size results [14] . In this work, we keep 
N explicitly finite and focus on the effect of the finite size on the response of the system to 
the time-dependent stimulus. 

Equilibrium states. - The two following order parameters are usually defined to describe 
the macrostate of the network: 

i 

which are the overlaps between each pattern and the neurons. 

However, these two parameters are not independent. A more convenient pair of magnitudes 
can be defined as follows: r is the fraction of bits where {si} coincides both with pattern 1 
and 2; p is the fraction of bits where {si} coincides with pattern 1 and differs from pattern 2. 
In Fig. ^ we have plotted a schematic representation of the two patterns and the quantities 
p and r. Taking into account that is the fraction of common bits minus the fraction of 
different bits between the network and pattern from Fig. ^ one immediately obtains: 

mi = 2r + 2p-l; m 2 = 2r + 2d - 2p - 1. (5) 

Also form this figure, it is easy to see that r and p are independent and can take on any value 
in the rectangle: r 6 [0, 1 — d], p £ [0,d]. Moreover, in terms of r and p, the free energy of the 
system for h = can be written as 



T = T r + T v (6) 



with 



d\ 2 1 f Nd 



F, - -1N[ P --) -fo S l Np ). (7) 



At zero temperature, the free energy has four minima, located at the corners of the available 
rectangle in the parameter space, i.e., r = 0, 1 — d, and p = 0, d. Two of the four equilibrium 
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Fig. 1 - Schematic representation of the two patterns and the state of the network. The upper part 
of the patterns represents the N(l — d) coincident bits between the patterns. Nr is the number of 
bits that the network has in common with both patterns, whereas p is number of bits in the network 
coinciding with pattern 1 and different from pattern 2. 

states, r = 1 — d, p = 0, d, exactly reproduce the two patterns. The other two minima, r = 0, 
p = 0, d, are the negatives of the stored patterns. Fig. a) shows a typical landscape of the 
free energy in the (r,p) plane for low temperatures (/3 = 2, d = 0.7). 

If we increase the temperature, the four minima shift to the middle point of the rectangle 
r = (1 — d)/2 and p — d/2. In the thermodynamic limit, the system undergoes two second- 
order phase transitions at /3 c , r = 1/(2 — 2d) and /3 C;P = l/(2d). In each transition the minima 
collide into either r = (1 — d)/2 or p = d/2, which corresponds, respectively, to a completely 
disordered state in the region of common and distinct bits between the two patterns (see 
Fig-D- The ability of the network to distinguish between the two patterns sensibly depends 
on which of the two transitions occurs first. 

If d < 0.5, i.e., if the two patterns share more than a half of the bits, (3 C . P > (3 c . r . Conse- 
quently, when the temperature increases from absolute zero the first transition occurs for the 
variable p, i.e., in the region of distinct bits. This means that for temperatures (3 s [Per, Pc,p], 
the system only exhibits two minima with p = d/2, i.e., with mi — mi [see Eq. |B|l]. One of 
these two minima approximately reproduces the common bits of the two patterns, whereas 
the distinct bits are completely disordered. The other minima is just the negative image of the 
first. Consequently, the system has mixed up the two patterns and it is unable to distinguish 
between them. The free energy landscape corresponding to this situation is plotted in Fig. [21 
b). 

On the other hand, if d > 0, 5, the two patterns are different enough to be distinguished 
even for intermediate temperatures. In this case /3 cr > (3 C . P , and the first transition occurs at 
(3 c ,r- Therefore, if (3 £ [P c . p , /3 c ,r], we have two minima with r = (l—d)/2, i.e., with mj = — m 2 . 
One of the two minima reproduces the distinct bits of pattern 1 and the other one the distinct 
bits of pattern 2. For both minima, the common bits are disordered. Although the system 
does not exactly reproduce the stored patterns, it perfectly distinguish between them. The 
free energy in this case is plotted in Fig.[5]c). 

Finally, above the maximum critical temperature, the only equilibrium state is completely 
disordered: r = (1 — d)/2 and p — d/2, or mi = m 2 = 0, as shown in Fig.|21d). 

Along this Letter we will focus only on the third case: patterns with d > 0.5 and tem- 
peratures corresponding to the landscape in Fig. c). The reason is that the system still 
distinguish between the two patterns, but we can reach temperatures high enough to clearly 
observe SSR. 
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Fig. 2 - The free energy landscape for h — and N = 10000. a) Low temperature, = 2, d = 0.7: 
The free energy presents four minima, corresponding to the two stored patterns and their respective 
negatives, b) Medium temperature, similar patterns, /3 = l,d = 0.3: There are two equilibrium states 
with p = d/2, i.e., with mi = rri2. One of the minima reproduces the common bits of the two patterns 
whereas the other one is its negative, c) Medium temperature, dissimilar patterns, /3 = 1, d = 0.7: 
There are two equilibrium states with r = (1 — d)/2, i.e., with mi = — mi- Each minima reproduces 
the distinct bits of each pattern, d) High temperature, j3 = 0.5, d = 0.7: The only minimum is the 
disordered state with r = (1 — d)/2, p — d/2, i.e., with mi = mi = 0. 

Results. - We have performed out-of-equilibrium Monte Carlo simulations [15] at inverse 
temperature (3 using the Hamiltonian The dynamics is defined by a standard Metropolis 
algorithm in which 100000 sweeps have been carried out averaging over 100 realizations for 
every parameter set. 

The simulations show a clear evidence of SSR. A relevant example is presented in Fig. |3 
with the distinctive features of stochastic resonance. For small N, fluctuations are strong and 
the system output is too noisy being unable to retrieve the patterns dynamically. The hoping 
between the attractors is random and not synchronized with the switches of the external 
stimulus. For very large N, fluctuations are weak and the system is quenched in a given 
attractor. However, for intermediate values of N, the system follows the oscillations of the 
external stimulus and the appropriate pattern for every half-period is retrieved very precisely. 
In order to obtain a more quantitative picture of SSR, we use the power spectrum S m (uj) 
of one of the order parameters. We will focus on the power spectrum of mi(t), but similar 
results are obtained if TOa(t) is chosen. A measure of the quality of the response of the system 
to the external input is the so-called signal amplification r\ [16], defined as the ratio between 
the power spectrum at the external frequency f2 = tt/T and the total power contained in the 
external stimulus: 

V = llm P5 ■ ( 8 ) 
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Fig. 3 - Time evolution of the order parameter mi showing the first evidence of system-size resonance. 
The solid thin line is the order parameter obtained from numerical simulations. The thick segments 
show the time intervals in which the external stimulus drives the system to retrieve one or the other 
pattern. The parameters values are: (3 = 1.2, d = 0.6, T = 100, h = 0.01. a) TV = 20; b) N = 60; c) 
N = 120. 



Following previous work [7, 16], we will use the signal amplification rj to locate and assess 
resonance phenomena. 

We have also obtained an analytical expression for S m (uj) reproducing quite well the nu- 
merical experiments. The main idea is to approximate the dynamics of the network by a 
two-state system and follow the calculation performed in [3, 17]. The details of the theory will 
be presented in a forthcoming publication [18]. Here we will only sketch the main steps of the 
calculation. 

The starting point is a master equation for the two-state system. The states are the 
two minima of the free energy, which are located numerically. As shown above, each of 
these minima reproduces fairly well the two stored patterns, hence we label them as 1 and 
2. The transition probabilities between these two states W\^ 2 and W 2 — >i depend on time 
as Wi-, 2 ([i(t)), n{t) being the pattern shown to the network at time t. We have chosen an 
Arrhenius-like expression as in [3,17]: 

Wi_v 2 (l) = cexp(-/3AF)exp(-/3hd 12 N/2) 

W^ 2 (2) = ce X p(-/5A/F)exp(/3M 12 iV/2), (9) 

where d± 2 is the difference between the values of the order parameter mi in the two minima, 
AJ- is the free energy barrier separating, at zero external field, the two minima along the path 
r = (1 — d)/2, with mi = —m 2 , and c is an arbitrary constant (see panel c in Fig. |2J). The 
transition probabilities Wi_>j are chosen to satisfy detailed balance: Wi_>2(l) = W 2 ^\(2). 

The corresponding master equation for this two-state stochastic process can be solved 
exactly to get the time-dependent moments < mi(t) > and < mi(t)mi(t + r) >. Finally, by 
using Wiener-Khinchin theorem [19], we obtain the following explicit expression for the power 
spectra S m (u>): 



1 - tanh 2 (/mdi 2 A0 ( 1 - tanh(^) 

\ TTVV l\l 

9d 2 W 2 



2W 



W 2 



7T ^ ' (2k-l) 2 (W 2 + {2k-l) 2 n 2 ) 

x (8[u - (2k - 1)Q] + S[lu + {2k + (10) 
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with 

W = Wi-> 2 (1) + 1Ti^ 2 (2) = cexp(-pAF)cosh(phdN). (11) 
The resonant behavior of the signal amplification r\ can be seen in Fig.0] There is a maximum 





Fig. 4 - The signal amplification, 77, versus the size of the system, TV for h = 0.01. The result of the 
numerical simulation is represented by dots while the analytical result is the solid line, a) d = 0.6, 
T = 500, P = 1.2. b) d = 0.75, T = 500, = 1.2. For larger values of d, the maximum is shifted 
to the left. The only free parameter in this theory is c, which has been chosen to fit best the Monte 
Carlo data. In a), c= 0.44; in b), c = 0.65. 

of the signal amplification at a finite size. The results of the numerical simulation (dots) 
correspond very well to the analytical results (solid line) given by Eq. i|l(J|) . The difference 
between both panels of Fig. 0] is exclusively due to a difference in the value of the Hamming 
distance d, keeping the other parameters f2, h and P equal. A larger value of d implies a 
higher energy barrier between the attractors which needs a larger noise intensity or smaller 
N to achieve the best resonance. Consequently, the maximum of 77 is shifted towards smaller 
N, and its maximum resonant value is reduced. Finally the dependence of 77 on the stimulus 




Fig. 5 - The behavior of 77 as a function of the frequency Q of the external stimulus shows the expected 
Lorentzian form, as in usual stochastic resonance. Again, the dots represent the numerical simulations 
and the solid line the analytical results. In this plot, d = 0.6, N = 50, /3 = 1.2 and h = 0.01. c=0.44, 
as in panel a) in Fig. 2] 



frequency f2 is shown in Fig.[S] ft shows the Lorentzian dependence expected in SR [3,20]. A 
good agreement between theory and numerical simulations is observed as well. 
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Conclusions. - In this paper we have presented numerical simulations and theoretical 
calculations, based on a two-state model, showing the presence of system size resonance effects 
in an attractor neural network. These effects are made evident by the resonant behavior of the 
signal amplification 77 as a function of the size of the system, as well as by the time evolution 
of the order parameters. We also point out the good agreement between analytical results 
and simulations. 

Our model shows that noise can provide the flexibility that an adaptive memory needs to 
follow a time-dependent stimulus. Since noise depends on the size of the system, we conclude 
that there is an optimal size for which there is maximal synchronization of the system to the 
evolving stimulus. We have explicitly shown this resonant phenomenon in a simple model 
with two attractors, but it is likely that more complicated models exhibit the same effect. 
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